System and Method for Assessing Data Quality During Clinical Trials

ABSTRACT

The present invention relates to a system and methods for facilitating the improvement of data quality generated during drug and medical device clinical trials. In one embodiment, the invention includes a system and method for generating an ideal subject suitability score for one or more interaction points of a clinical trial. One or more individual subject suitability scores may then be generated from actual subject interaction with the clinical trial for each of the interaction points of the clinical trial. For each interaction point for which an ideal subject suitability score is generated, the one or more individual subject suitability scores may be compared to the ideal subject suitability score to determine the deviation there between. The quality of data collected from actual subject interaction with the clinical trial may be a function of the difference between the individual and ideal subject suitability scores.

RELATED APPLICATION

This application is a continuation application of U.S. application Ser.No. 14/212,467, filed on Mar. 14, 2014; which in turn is a continuationapplication of U.S. application Ser. No. 11/577,681, filed on Dec. 12,2007, now U.S. Pat. No. 8,682,685; which is a 371 National Stage Entryof International Patent Application No. PCT/US2006/04542, filed on Feb.10, 2006; which claims priority from U.S. Provisional Patent ApplicationNo. 60/657,386, filed Mar. 2, 2005. Each of the foregoing applicationsis incorporated herein by reference in its entirety.

FIELD OF THE INVENTION

The invention relates to a system and method for assessing andfacilitating the improvement of data quality generated during drug andmedical device clinical trials.

BACKGROUND OF THE INVENTION

The development of a new drug or medical device can be a challenging andtime consuming process. If preclinical testing suggests that a promisingcompound might be well tolerated in humans, it may be tested for safetyand pharmacokinetics (drug absorption and metabolism) in healthyvolunteers (Phase I). If the results of Phase I trials warrant furtherinvestigation, a limited number of patients with the target disease maybe challenged with the drug under carefully controlled conditions toevaluate its efficacy and further establish safety and proper dosages(Phase II). If these trials are successful, the drug enters large-scaletrials to better characterize its safety and efficacy in patients (PhaseIII). Typically, clinical trials are coordinated by either contractresearch organizations (CROs) or academic medical centers that aresponsored by the pharmaceutical manufacturer. Physicians at theseinstitutions conduct the clinical trials and care for thepatients/subjects. The Food and Drug Administration (FDA) is theregulatory body having oversight of drug development, which encompassesthe preclinical and clinical trial phases of the new drug discovery andtesting in humans.

A significant portion of the time and expense of conducting clinicaltrials arises from the need to assure that the resulting data isaccurate. Patients are selected, treated, and evaluated by a meticulousprotocol, and the results are usually recorded on standardized forms(case report forms or CRFs) that are collected and analyzed by thesponsor or its designee. To ensure the validity and accuracy of thedata, the pharmaceutical company periodically sends a monitor to studysites to verify that patients are treated according to the studyprotocol and that the information is reported according to the studyprotocol. Monitoring alone can represent up to 30 percent of the costsof a clinical trial. Most pharmaceutical companies also have separatequality assurance departments to review forms and audit data and safetydepartments to monitor and prepare reports on adverse events.

From the pharmaceutical manufacturer's perspective, the key issues withrespect to data quality and integrity may include how to accuratelycollect the information that is necessary to assess the safety andeffectiveness of the experimental therapy, as well as how to ensure thequality and integrity of that information, while controlling costs andreducing the time consumed by the clinical trial process. From the FDA'sperspective, however, the key issue is ensuring that data submitted insupport of an application is a valid representation of the clinicaltrial, especially as the data pertains to drug safety, pharmacokinetics,and efficacy.

Under the Federal Food, Drug, and Cosmetic Act, pharmaceuticalmanufacturers must obtain a research or marketing permit beforebeginning studies on certain commodities such as new human drugs,medical devices, veterinary drugs, and food additives. FDA approvesthese permits, and also regulates biomedical research whose results arethen submitted in support of an application for such a permit. The FDAhas two principle objectives in regulating this research: 1) to protectthe rights and welfare of human research subjects, and 2) to assure thequality and integrity of the biomedical research data used to supportthe initiation or expansion of clinical trials, the approval of newproducts and indications, and the labeling of these products.

Pharmaceutical companies monitor and audit clinical trial data: 1) toensure the safety of the human subjects, 2) to ensure that the company'sinvestment results in a marketable product, and 3) because it isrequired by the FDA as follows:

-   -   Sponsors are responsible for selecting qualified investigators,        providing them with the information they need to conduct an        investigation properly, ensuring proper monitoring of the        investigation(s), ensuring that the investigation(s) is        conducted in accordance with the general investigational plan        and protocols contained in the IND (Investigational New Drug        application), maintaining an effective IND with respect to the        investigations, and ensuring that FDA and all participating        investigators are promptly informed of significant new adverse        effects or risks with respect to the drug . . . 21 C.F.R.        31.250.

Although each company may structure its activities in different ways,responsibility for monitoring is typically distributed as follows. Theclinical research department includes medical monitors, often physicianswith a considerable amount of clinical experience. The greater burden ofmonitoring falls to the clinical research associates, who go into thefield to make sure that sites are properly initiated and the data arecollected appropriately. Most companies also have a separate clinicalquality assurance department that conducts in-house file audits toensure that protocols are written correctly, and conducts site andinvestigator audits to confirm the qualifications of the investigator,to match case report forms with patient charts, and to review theinformed consent forms. Members of the biostatistics and data managementgroup, which is usually separate from the clinical research group,monitor all the data received from the field and investigate emergingtrends that might affect safety. The drug safety department collectsdata on serious adverse effects. Finally, the regulatory affairs groupcompiles expedited serious adverse effects reports and sends them to theappropriate regulatory agencies.

This process generates an enormous volume of data, and the greater theamount of data that is collected the higher the probability of error.The subsequent task of reconciling the various data streams becomes moredifficult.

The reality in clinical drug development is that a clinical trial isonly as good as the quality of the data. Under current standards ofpractice, some of which are regulated, “monitoring” represents anability to assess study progress but not the quality of the data. Thepharmaceutical industry's concept of data quality relates to data entryissues rather than the intrinsic value of the data, i.e., generatingexperimental results that meet the objectives of the trial. Thisrepresents a major drawback in how clinical trials are managed, giventime sensitive issues (e.g., patent life) and financial issues (cost ofdevelopment and recovery of the investment with product launch) thatdrive this effort. Refocusing the clinical trial development effort toprospectively evaluate the quality of the data without biasing theoutcome would provide both time and cost savings.

The protocol of a clinical trial represents a complicated roadmap fortrial study. Some studies are of a sufficiently long duration such thatthey may take months to years to complete. The data that is generatedfor each of the multiple interaction points in a complicated protocol isfundamental to the successful completion of a clinical trial. Thedefinition of a successful study is one that fulfills the experimentalobjectives of the protocol, i.e., accurately portrays the safety andefficacy of a new drug or medical device (or the lack thereof). Althoughthe clinical trial protocol may have clear experimental objectives, itis the execution of the protocol which can be variable. Late phasestudies or pivotal studies (e.g., Phase 3) can often require 500 to10,000 study subjects, depending on the indication, to demonstratesafety and efficacy. Generally, this requires multiple investigatorsand/or study sites to enroll and complete this large number of subjects.In general, the FDA requires that a sponsor of a drug successfullycomplete two pivotal trials of sufficient magnitude to demonstratesafety and efficacy. As one can imagine, given the challenges of studiesof this size, there may be a great variation of interpretation in howstudy subjects are entered and “complete” the protocol. Thesediscrepancies lead to the generation of data of poor quality anduniversally delay all clinical drug development programs.

Typically, as a clinical trial goes forward, subject data is collectedfrom a primary source, such as a patient's chart, and transferred tocase report forms or other subject data receptacle. At this point,mandated monitoring normally takes place, and upon study subject and/orstudy completion, the data is entered into a database or other datastorage area. In current practice, no attempt is made to assess thevalue or quality of the collected information until the data is enteredinto the database alongside various predetermined acceptable ranges foreach data variable. Data that is outside the range for each variable maytrigger a “data query” to the respective study site for purposes ofresolution. If there are enough data queries for a subject that cannotbe appropriately resolved, the subject may be invalidated and thesubject's data may not be used in the study. Depending on the overallstudy design, subjects that are invalidated often need to be replaced tomeet sample size requirements. This failure to proactively assess dataquality while the study is in progress results in budget and timelinedeviations because additional subjects need to be obtained and theirdata collected and analyzed.

These and other problems exist.

SUMMARY OF THE INVENTION

The invention overcomes these and other drawbacks in the management ofclinical development programs for new drugs or devices. One aspect ofthe invention relates to the analysis of the principal components of aclinical study protocol which defines 1) the characteristics of a studysubject, 2) the procedures that, if implemented correctly, yield validand high quality data, and, if desired, other components. Another aspectof the invention relates to the unbiased, real-time review of the dataproduced during a clinical trial. The invention enables evaluation ofdata quality as it relates to achieving the clinical objective detailedin the study protocol. An unbiased review indicates that the blind isnot broken in a typical double blind, randomized, placebo controlledtrial. Quality of the data in this context may include the intrinsicvalue of the data.

The invention develops a framework or indexing system to assess dataquality in a clinical trial, i.e., the intrinsic value of the data inmeeting the experimental results that fulfill the objectives of thestudy, on an ongoing basis. It goes beyond simply evaluating the actualvalue of the particular variable, whether it is discrete (e.g., yes orno) or continuous data. It evaluates the data quality in the context ofthe protocol.

In one embodiment, the invention provides a method for assessing dataquality of data generated during a clinical trial. In one embodiment,this method includes developing an ideal subject suitability score forone or more interaction points of a clinical trial. The ideal subjectsuitability score for each interaction point represents a numericalvalue that defines data of ideal quality for each interaction point ofthe clinical trial protocol. The ideal subject suitability score may beused as a benchmark for the evaluation of actual patients that areenrolled in the study.

An ideal subject suitability score for an interaction point may begenerated by defining data factors that represent the quality variablesfor the interaction point detailed in the clinical trial protocol. Ascoring system may then be associated with each defined data factor. Thescoring system may yield an “ideal data factor score” when applied to an“ideal data value” for each data factor. An ideal data value may includea value that signifies ideal data quality for the respective datafactor. In some embodiments, ideal data factor scores for each datafactor may be weighted according their importance to data quality. Theseideal data factor scores may then be utilized to determine the idealsubject suitability score for the interaction point. In one embodiment,the ideal subject suitability score may be an arithmetic sum of weightedideal data factor scores for the interaction point. In some embodiments,other methods may be used to determine the ideal subject suitabilityscore from the ideal data factor scores.

For each of one or more subjects participating in the clinical trial, an“individual subject suitability score” may be developed for eachinteraction point of the clinical trial. This individual subjectsuitability score is based on an individual subject's interaction withthe interaction point (e.g., points at which data is recorded from apatient during the clinical trial). Subject interaction with theinteraction point may produce “actual data values” for each data factoridentified for the interaction point. For each data factor, the scoringsystem associated with that data factor may be applied to the actualdata value, yielding an “actual data factor score.” In some embodiments,the actual data factor scores may be weighted according to theirimportance to data quality. The actual data factor scores may then beutilized to determine the individual subject suitability score for aparticular subject's interaction with the interaction point. In oneembodiment, the individual subject suitability score may be anarithmetic sum of the weighted actual data factor scores for a subject'sinteraction with the interaction point. In some embodiments, othermethods may be used to determine an individual subject suitability scorefrom the actual data factor scores.

The individual subject suitability score and the ideal subjectsuitability score may be used to assess the quality of the data from theinteraction point. For example, if upon review, an actual subject'sindividual subject suitability score significantly deviates from theideal subject suitability score for an interaction point, it may be areal-time “early warning” that data quality is low, that the clinicaltrial protocol is not being executed properly, and/or other problems.

In another embodiment, the invention provides a method to determinewhether a prospective clinical trial site may be suitable for use in anupcoming clinical trial. This method may utilize an ideal subjectsuitability score representing a study subject having attributes thatare likely to produce high quality data. In one embodiment, actual dataregarding the prospective study site may be collected such as, forexample, historical data of previous trials conducted at the site (whichmay be indicative of the site's ability to enroll subjects that producehigh quality data), population data of the area (which may be indicativeof the presence of subjects capable of producing high quality data forthe particular clinical trial at issue), and/or other data. This actualdata may be utilized to generate a study site aptitude score, which maybe compared to the ideal subject suitability score to determine whetherthe prospective clinical trial site is suitable for use in the upcomingclinical trial.

In another embodiment, the invention provides a method to rapidlydetermine the acceptability of a potential subject as a candidate forstudy entry, prior to the potential subject receiving therapy. Thismethod may utilize an ideal subject suitability score that may representan ideal clinical trial subject having attributes that are likely toproduce high quality data. This ideal subject suitability score may begenerated by evaluating the key variables in the protocol that arespecific to determining subject enrollment to determine data factors,applying scoring systems to these data factors to produce data factorscores, and using these data factor scores to generate the ideal subjectsuitability score. The scoring systems may then be applied to actualdata values gathered from potential clinical trial subjects, to produceindividual subject suitability scores. In one embodiment, comparison ofthe ideal subject suitability score with an individual subjectsuitability score may indicate a subject's acceptability as a clinicaltrial candidate.

In another embodiment, the invention provides a method to evaluate theelements of an unsuccessful or otherwise completed clinical trial inorder to understand the processes that may have contributed to itsfailure (or for evaluating other characteristics). This may beaccomplished by generating ideal subject suitability scores for one ormore interaction points of a completed clinical trial, retrospectivelygenerating individual subject suitability scores for trial subjectsbased on available data, and identifying what elements of the clinicaltrial contributed to its failure.

In another embodiment, the invention provides computer-implementedmethods, a computer-implemented system, and/or a computer readablemedium for performing and enabling the features, functions, and methodsdescribed herein.

These and other objects, features, and advantages of the invention willbe apparent through the detailed description of the preferredembodiments and the drawings attached hereto. It is also to beunderstood that both the foregoing general description and the followingdetailed description are exemplary and not restrictive of the scope ofthe invention.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is an exemplary illustration of a process for assessing clinicaltrial data quality according to an embodiment of the invention.

FIG. 2 is an exemplary illustration of a process for generating an idealsubject suitability score according to an embodiment of the invention.

FIG. 3 is an exemplary illustration of a process for generatingindividual subject suitability scores according to an embodiment of theinvention.

FIG. 4 is an exemplary bar graph showing category defined ideal subjectsuitability scores by visit.

FIG. 5 is an exemplary graph showing aggregate subject suitability scoreby visit of an ideal subject (Ideal SSS) versus ten actual subjects(SSS-1 to SSS-5).

FIG. 6 is an exemplary bar graph showing category defined subjectsuitability score enrollment visit of ideal versus actual subjects.

FIG. 7 is an exemplary illustration of a process for evaluating apotential clinical trial site according to an embodiment of theinvention.

FIG. 8 is an exemplary illustration of a process for evaluating subjectsuitability according to an embodiment of the invention.

FIG. 9 is an exemplary illustration of a process for evaluating acompleted clinical trial according to an embodiment of the invention.

FIG. 10 is an exemplary computer-implemented system according to anembodiment of the invention.

DETAILED DESCRIPTION OF THE INVENTION

The invention provides a system and method for critically assessing thequality of data generated during clinical trials while they are inprogress. Quality of data in this context may include the intrinsicvalue of the data. In one embodiment, the invention enables a user orusers to assess the progression of a clinical trial or a clinicaldevelopment program. In one embodiment, this real-time, proactiveassessment of clinical trial progress may include methods thatcritically review the clinical protocol and accurately define the datafactors that govern the production of high quality data. In anotherembodiment, the invention may forecast the ability of clinical trialinvestigators or study sites to successfully perform a clinical trial.

Every clinical trial has a protocol which specifies the exact timing andnature of the measurements and/or intervention (e.g., “data collection”)to be performed on each patient. A clinical trial protocol may includeand/or define a series of “interaction points.” An interaction point mayinclude any of the visits or other events within a clinical trial wheredata is collected from a study subject. A clinical trial subject or“subject,” as used herein, may include a human patient or othermammalian subject. While an interaction point may include a subjectvisit to a hospital or other medical facility for the purposes ofadministering treatment, collecting data, or other interaction, aninteraction point may also take place in other locations or environments(e.g., at the subject's home) and may include any interaction that thesubject has with the clinical trial. The timeline of a clinical trialmay begin with the screening and enrollment of subjects (the firstinteraction point) and may continue with one or more subject visits(subsequent interaction points).

The clinical protocol that is written to assess the safety and efficacyof a new drug for a specific disease or indication, in essence,describes a method by which subjects are selected, drug or placebo isadministered, safety and efficacy measurements are made and analyzed,and regulatory obligations are met. It is these parameters that definethe data that is to be collected from each of the interaction points. Assuch, data collected for this invention may not only include patientresponse or physiological reaction to the administration of a treatment,but may include data collected regarding treatment administration (e.g.,circumstances of drug administration, etc.), or other data.

Viewed on a strictly operational level, in clinical trial deployment,implementation, and completion, there is an inherent process and a finalproduct: the data. The clinical trial protocol defines a process and aspecification for the final product, similar in principle to themanufacture of any finished good. For example, the protocol may containinclusion and exclusion criteria that define the characteristics ofsuitable study subjects. The protocol may also define various events andprocesses for the enrolled subject, including drug administration andclinical or laboratory testing, etc. The end-product of the clinicaltrial is safety and efficacy data that is generated to answer theexperimental questions as defined in the protocol. This safety andefficacy data is usually then recorded in case report forms or in otherformats. It is the quality or intrinsic value of this safety, efficacy,or other data product that the invention is concerned with.

According to an embodiment of the invention illustrated in FIG. 1, amethod 100 for assessing the quality of clinical trial data is provided.In an operation 101, a review and/or analysis of a clinical trialprotocol and/or case report forms may be undertaken to generate an“ideal subject suitability score.” The ideal subject suitability scoremay represent a benchmark for clinical trial data quality when all thedata is collected correctly. Each particular interaction point of aclinical trial may have its own ideal subject suitability score, againstwhich actual subject data collected for that interaction point ismeasured. In some embodiments, some or all interaction points for aclinical trial may be grouped and an ideal subject suitability score maybe developed for the group or groups.

The ideal subject suitability score for a particular interaction pointmay be based at least in part upon the data requirements specified inthe trial protocol for that study visit. For example, during the firstinteraction point of a clinical trial (e.g., the enrollment visit), thespecification for the ideal subject suitability score may be determinedby the criteria listed in the inclusion and exclusion criteria and thedata required for the outcome and safety variables. For subsequentinteraction points, the ideal subject suitability score may be based onthe detailed study collection needs as found in the protocol (in manyclinical trial protocols this information can be categorized as safetyand efficacy data).

FIG. 2 is an exemplary illustration of a process 200 according to anembodiment of the invention, wherein an ideal subject suitability scorefor a given interaction point may be generated (as in operation 101 ofFIG. 1). In an operation 201, the “data factors” that are to be used forthe ideal subject suitability score for the given interaction point maybe identified from the clinical trial protocol. In identifying the datafactors for an interaction point, the procedures and events involved insubject interaction with the clinical trial protocol may be evaluated.These procedures and events may include enrollment (wherein inclusionand exclusion criteria are utilized), administration of therapy (e.g.,administration of drugs, or placebo), observation, clinical and/orlaboratory testing (wherein safety and efficacy criteria are utilized),and/or other procedures and events. The quality variables involved withthese procedures or events (organized by interaction point) may beutilized to define the “data factors” for an interaction point. The datafactors defined for each interaction point may vary depending on theprocedures prescribed by the clinical trial protocol.

In some embodiments, multiple data factors may be identified and appliedfor a procedure or event within an interaction point. These multipledata factors may be used, for example, with procedures or events relatedto important safety and efficacy measurements. For example, in a trialthat involves generation of continuous (as opposed to discrete data,e.g., yes or no) data such as, for example, data derived from clinicallaboratory testing, ratings of pathology samples or from endoscopicprocedures, various body scans [x-ray, PET scans, MRI, etc.],questionnaires which yield an aggregate score, or other continuous data,multiple data factors that quantify the quality of each of the variablespresent in the continuous data may be used. These multiple data factorsmay, for example, correspond to the conditions delineated in theprotocol for how measurements are made. For example, for blood tests,the timing of a blood sample relative to administration of adrug/placebo may be an additional factor that is collected (additionalto the actual measurement of drug concentration in the blood). Otheradditional factors may exist such as, for example, were theprotocol-specific requirements for testing met, were the samples handledcorrectly, were appropriately trained personnel involved, or otherfactors.

In an operation 203, the identified data factors for an individualinteraction point may be indexed and/or categorized according to theirrespective characteristics (e.g., do they deal with inclusion/exclusion,safety, efficacy, or other characteristics; what specific subjectinteraction procedure or event [e.g., drug concentration measurement]are they associated with). As discussed in detail below, this indexingmay aid in selective evaluation of the various components thatcontribute to overall data quality. In an operation 205, the variousdata factors may be ranked in order of importance within the giveninteraction point (or their importance within the subcategoriescomprising the interaction point, e.g., inclusion/exclusion, safety,efficacy, etc.). In an operation 207, each of the identified datafactors for the interaction point may be associated with a scoringsystem. Different scoring systems may be utilized for different datafactors. The scoring system may produce a “data factor score,” whenapplied to a data value for a data factor. Examples of data values mayinclude the absence or presence of a condition (e.g., is the persontaking a sample properly trained?), a numerical value regarding subjectinteraction with the interaction point (e.g., the time period elapsedbetween a measurement and administration of treatment), or other values.

In an operation 209, the scoring system for each data factor may then beapplied to an ideal data value for the data factor to produce an idealdata factor score. Because process 200 deals with generating an idealsubject suitability score, an “ideal data value” indicating the highestpossible data quality for the data factor is used. In some embodiments,a range of values may be considered ideal data values. In theseembodiments, any data value from the range may be used.

In an operation 211, each ideal data factor score may then be weightedto signify its relative importance to data quality for the giveninteraction point. Some of the variables represented by the identifieddata factors may be more indicative of data quality than others, thus,their corresponding data factor scores may be weighted more heavily.Weighting of the data factor score may be dependent on the clinicaltrial protocol. For example, in some embodiments, a data factor forefficacy may be more heavily weighted for data values representing anon-time measurement, and decrements in data factor score may be moresignificant. In some embodiments, scoring systems may includeexponential or other weighted decrements in data factor score forprotocol deviations.

In an operation 213, an ideal subject suitability score may bedetermined based on the ideal data factor scores. In one embodiment, anarithmetic sum of weighted ideal data factor scores for a giveninteraction point may be used to generate the ideal subject suitabilityscore for the interaction point. Other methods of determining idealsubject suitability scores from ideal data factor scores may be used.The ideal subject suitability score represents the benchmark againstwhich actual subject data quality may be measured for a specificinteraction point.

An example of application of a scoring system in obtaining an idealsubject suitability score may be as follows. For a particularinteraction point of a clinical trial, the clinical trial protocol maydictate that a blood sample is required exactly 24 hours afteradministration of a drug. This blood sample may be utilized to measurethe concentration of the drug in the subject's system 24 hours afterdrug administration. In developing the ideal subject suitability scorefor this interaction point, a “timing” data factor for this bloodcollection and drug concentration measurement (there may be many otherfactors for this collection and measurement) may be identified andassociated with a scoring system (as in operation 207). This scoringsystem may dictate that an ideal data value for this data factor is anelapsed time of 24 hours after drug administration. The scoring systemmay also dictate that this ideal data value of 24 hours yields a datafactor score of 10 and that for each hour deviation (plus or minus) froma precise data value of 24 hours, 0.5 points will be deducted from theperfect data factor score of 10. Therefore, the timing data factor forthis blood sample collection may yield an ideal data factor score of 10.This ideal data factor score may then be used (alone or with other idealdata factor scores) to determine the individual subject suitabilityscore.

Referring back to FIG. 1, an operation 103 may be utilized to generateone or more individual subject suitability scores for a subjectparticipating in the clinical trial. Methods similar to those used togenerate the ideal subject suitability score for a particularinteraction point may be used to generate individual subject suitabilityscores for that interaction point. These methods may also be based onthe clinical trial protocol. FIG. 3 illustrates an exemplary process300, wherein an individual subject suitability score may be developedfor a specific interaction point of a clinical trial (as in operation103 of FIG. 1). In an operation 301, actual subjectparticipation/interaction with the procedures or events of theinteraction point may be used to produce “actual data values” for eachof the one or more data factors that have been identified from theclinical trial protocol for the interaction point. In one embodiment,the actual data values for each data factor may be collected andrecorded.

In an operation 303, the actual data values for each data factor maythen be indexed according to the index developed for the ideal subjectsuitability score of the interaction point. In an operation 305, theactual data values may be ranked according to the ranking systemdeveloped for the data factors of the ideal subject suitability score.In an operation 307, the scoring system associated with each data factormay be applied to the actual data value for that data factor. Applyingthis scoring system may produce “actual data factor scores.” In anoperation 309, the actual data factor scores may then be weightedaccording to the same weighting system that were used for the datafactors in the calculation of the ideal subject suitability score forthe interaction point.

In an operation 311, the individual subject suitability score may thenbe determined from the actual data factor scores for an individualsubject's interaction with the interaction point. The individual subjectsuitability score represents a measure of the quality of data obtainedfor an individual clinical trial subject for the interaction point. Insome embodiments, an arithmetic sum of weighted actual data factorscores may be utilized to determine the individual subject suitabilityscore. Other methods of determining individual subject suitabilityscores from actual data factor scores may be used. However, the methodused to determine an ideal subject suitability score for an interactionpoint, will parallel the method used to determine the individual subjectsuitability scores for the interaction point (as they both are derivedfrom the same clinical trial protocol).

An example of applying a scoring system to actual data values may beseen by revisiting the blood collection example given above. An actualdata value for the timing data factor described above may equal 24 hours(i.e., blood collection 24 hours after drug administration). Under thescoring system described above, this actual data value of 24 hours willyield a perfect actual data factor score of 10. However, if the bloodsample was collected from a subject 25 hours after administration of thedrug, the actual data value for the subject would be 25 hours. Under thescoring system, a 25 hour actual data value will yield a 9.5 actual datafactor score (because of the 0.5 deduction in data factor score forevery hour deviation from the ideal data value). This actual data valuescore of 9.5 may be used (alone or with other actual data factor values)to determine the subject's individual subject suitability score. In aninstance in which no blood was drawn for this measurement, the actualdata value for this data factor may be zero. Therefore, applying thescoring system to an actual data value may yield an actual data factorscore of zero.

In an operation 105, a data quality assessment may be generated for aninteraction point using one or more of the individual subjectsuitability scores for the interaction point and the ideal subjectsuitability score for the interaction point. This data qualityassessment may reflect the data quality of a single clinical trialsubject (if only one individual score was used) or of many subjects (ifmultiple individual scores were averaged and used). In some embodimentsthe data quality assessment may be generated by comparing the one ormore individual subject suitability scores of an interaction point tothe ideal subject suitability score of the interaction point. In someembodiments the deviation of an individual subject suitability scorefrom an ideal subject suitability score may be indicative of dataquality of the data corresponding to the individual score. In someembodiments, the greater the individual score deviates from the idealscore, the poorer the quality of the data.

This data quality assessment enables real-time, proactive monitoring andassessment of the quality of the data collected from the individualsubjects participating in the trial. This monitoring and assessmentenables timely remediation or other measures to be taken regarding theinvestigators, the study sites, the protocol, and/or the clinical trialas a whole.

In some embodiments, individual and ideal subject suitability scores maybe utilized for one or more data mining processes. For example, in someembodiments (as described above), the quality of data for individualsubjects at individual interaction points may be assessed by comparingan individual subject suitability score of a subject at a particularinteraction point to the ideal subject suitability score for thatinteraction point. In other embodiments, individual subject suitabilityscores may be aggregated and/or averaged and compared to the idealsubject suitability score for a particular interaction point. This mayenable assessment of the data quality of some or all subjects in aclinical trial for that interaction point. In some embodiments,interaction points may be grouped and ideal and individual subjectsuitability scores may be used to assess the quality of data resultingfrom some or all interaction points in a clinical trial. In someembodiments, individual subject suitability scores may be grouped bytrial site to measure a particular site's data quality. Other selectivedata categorizations, assessments an/or data mining may be performed.

In an operation 107, after data quality for the data of interest hasbeen measured, remedial measures may be generated and/or performed.Remedial measures may include, for example, adjusting the clinical trialprotocol, re-training personnel who collect data (or other personsinvolved in conducting the clinical trial), shutting down a study site,shutting down the trial altogether, or other measures.

Assessing data quality according to the invention may not necessarilyinvolve judgment of the actual measurements obtained from subjectinteraction with the clinical trial. The methods of the invention do notevaluate the fact that a measurement may represent good or bad news tothe sponsor of the clinical trial (e.g., too much, too little, or justenough of a physiological response in the patient's system). However,if, for example, a sample was taken too early or too late, the actualmeasurement data may be of poor quality and thus of little importance tothe clinical trial because the measurement was obtained through adeviation from the clinical trial protocol. The methods described hereinassess this data quality.

One of the problematic issues that affects clinical trials may includesubject-specific quality issues. Subject-specific quality issuesroutinely include whether the subjects enrolled in the study aresuitable for study entry. The invention may aid in the alleviation ofthese subject-specific issues.

The specification for subject suitability may be detailed by theinclusion and exclusion criteria for the particular trial. This may be aliteral checklist of characteristics that are appropriate (inclusioncriteria) or inappropriate (exclusion criteria) for subjects in theclinical trial. An ideal subject may receive all affirmatives forinclusion and all negatives for exclusions. Subjects that do not meetthis ideal specification may not be suitable for study entry. This maybe where the problem in clinical trial data quality begins.

To help illustrate the subject-specific quality issues, an example ofthe inclusion/exclusion criteria for a drug trial is detailed below:

A Randomized, Multi-Center, 8 Week, Double-Blind, Placebo-Controlled,Flexible-Dose Study to Evaluate the Efficacy and Safety of Drug X inChildren and Adolescents with Major Depressive Disorder (MDD) InclusionCriteria:

-   -   Male or female patients age 7 years 0 months to 17 years 11        months inclusive.        -   Yes □ No □    -   Diagnosis of MDD, either single episode or recurrent according        to DSM-IV (296.2 or 296.3, respectively) confirmed by the        Kiddie-Sads-Present and Lifetime version (K-SADS-PL)        semi-structured interview        -   Yes □ No □    -   Patients with a total raw summary score of 45 or greater on the        Children's Depressive Rating Scale-Revised (CDRS-R) at the        Screening and Baseline Visits.        -   Yes □ No □    -   Custodial parent's or legal guardian's written informed consent        before performance of any study-specific procedures and        patient's assent and/or consent where required.        -   Yes □ No □

Exclusion Criteria:

-   -   Patients who in the investigator's judgment present with a        clinically predominant Axis I disorder other than MDD.        -   Yes □ No □    -   Patients with any history of psychotic episode or psychotic        disorder.        -   Yes □ No □    -   Patients with a history of Bipolar Disorder.        -   Yes □ No □    -   Patients with mental retardation.        -   Yes □ No □    -   Patients diagnosed with Substance Abuse or Dependence within 3        months prior to screening.        -   Yes □ No □    -   Patients who tested positive for illicit drug use at the        Screening visit.        -   Yes □ No □    -   Patients who, in the investigator's judgment, posed a suicidal        or homicidal risk.        -   Yes □ No □    -   Patients who have taken other psychoactive drugs with in the        time frames specified below prior to the screening visit:    -   Fluoxetine, MAOIs—4 weeks or less        -   Yes □ No □    -   Depot antipsychotics—12 weeks or less        -   Yes □ No □    -   Antidepressants other that MAOIs or fluoxetine, etc. —14 days or        less.        -   Yes □ No □    -   Hypnotics, benzodiazepines, and all other sedatives (including        sedating antihistamines)—5 half-lives or 14 days (whichever is        longer) or less.        -   Yes □ No □    -   Any CNS-active herbal/natural supplement or preparation known or        thought to have any psychoactive effects—14 days or less        -   Yes □ No □    -   Patients with epilepsy.        -   Yes □ No □    -   Patients who, in the opinion of the investigator, would be        non-compliant with the visit schedule or other study procedures        -   Yes □ No □    -   Patients with clinically significant abnormalities in        hematology, blood chemistry, ECG, or physical examination at        Screening that was not resolved by the Baseline visit.        -   Yes □ No □    -   Patients with known hypersensitivity to SSRIs        -   Yes □ No □    -   Patients who had electroconvulsive therapy with 3 months of        Screening        -   Yes □ No □    -   Female patients who had a positive serum HCG pregnancy test or        who were lactating Yes □ No □    -   Sexually active female patients who were not using a reliable        method of contraception        -   Yes □ No □    -   Patients who received any investigational drug within 6 months        of Screening        -   Yes □ No □    -   Patients requiring concurrent psychotherapy.        -   Yes □ No □    -   Patients who, in the judgment of the investigator, had a clear        history of non-response to SSRI treatment for their MDD, defined        as a non-response to at least two different SSRIs following        adequate courses of treatment (i.e., received recommended doses        for 4 to 6 weeks for each)        -   Yes □ No □

Since this exemplary study is a study of the effect of treatment of anexperimental drug on major depressive disorder (MDD) in children andadolescents, it may be essential that each proposed subject meet theseinclusion criteria. Therefore, the subjects must be age appropriate,have a diagnosis of MDD, and meet the stringent criteria set by usingthe defined method of measuring depression and having a score as definedin the protocol. In an embodiment of the invention, the data factors (ortheir resultant data factor scores) for these criteria may be heavilyweighted. In this example, the inclusion requirement for parental orguardian consent may be essential for legal purposes. However, inregards to scoring under the present invention, this requirement may notbe weighted as heavily as the three other inclusion criteria, because itmay be thought to have little effect on the quality of the subsequentdata collected.

In regards to exclusion criteria, all of the 23 requirements barringentry to the study listed above are important; however, some may be moresignificant than others, and thus, may be weighted more than others. Anypatient characteristic that will impact negatively on the trial and thatis met by the subject will cause the individual subject suitabilityscore to drop. In this example, subjects with increased risk for suicidewho have taken drugs that may interfere with the experimental therapywithin a defined time period will cause a negative deflection on thosesubjects' individual subject suitability scores for the enrollmentinteraction point. Accordingly, subjects who have a concomitantcondition which may affect study results will also cause a negativedeflection on those subjects' individual subject suitability score forthe enrollment interaction point. It may be determined that thesecharacteristics will be weighted more negatively in the subjects'individual subject suitability scores since these conditions may havebeen determined to substantially decrease data quality through theirunique negative impact on accuracy of the collected data.

Other issues that affect the quality of clinical trial data areoperational issues. These operational problems revolve around the actualimplementation of a clinical trial's protocol. The invention may aid inassessment and resolution of these operational issues.

A typical protocol may describe exactly when subjects receive treatmentsand the battery of safety and efficacy testing they undergo before,concurrent to, and/or after such treatments. The uniformity of theseprocedures is important since hundreds of subjects are studied atmultiple study sites, and successful measurement of the effects of anexperimental drug versus placebo depends on proper execution of trialprotocol. An example of evaluation criteria (safety and efficacymeasures) for a given protocol is found below:

Evaluation Criteria

-   -   Efficacy Parameters:        -   The primary efficacy variable is the change from baseline in            the CDRS-R total score. The score will be assessed baseline            and four and eight weeks after randomization to therapy.        -   The secondary efficacy variable is the change from baseline            in the Clinical Global Impression (CGI) Severity of Illness            item score. The score will be assessed baseline and four and            eight weeks after randomization to therapy.    -   Safety Parameters:        -   Safety will be assessed by 1) adverse event monitoring, 2)            vital signs, 3) laboratory evaluations, 4) serum pregnancy            tests, 5) ECGs, and 6) physical examinations. Measurements            will be made at baseline and four and eight weeks after            randomization to therapy.

The goal in this exemplary trial is to demonstrate that the experimentaldrug is useful in treating MDD in pediatric and adolescent subjects, andthat there are no unacceptable safety risks. The efficacy measure ismade by using a standardized assessment tool that is used to measuredepression before, during, and after treatment. The safety determinationis made using a standard battery of tests that are performed before,during and after treatment. To assess the quality of the data withoutbiasing the outcome of the trial, the method of the present inventionlooks at the particular data points that need to be collected, but doesnot evaluate the data in the larger picture, i.e., is the drug safe andeffective.

For example, the primary measurement of efficacy (e.g., the primaryefficacy variable) for this study is the CDRS-R total score, which needsto be collected at specific time points after the drug has beenadministered, i.e., four and eight weeks into the study. To assess thequality of this important variable, the following questions may bedevised and utilized to develop data factors for each of the variousinteraction points of the trial: 1) Was the CDRS-R assessment tooladministered by the designated individual trained in using the test? 2)Was the measurement successfully completed? 3) Was the data collected?4) Was the data collected within the correct time interval as specifiedby the protocol? 5) Is the resulting data within the expected range forthe particular variable(s)? All five of these data factors illustratehow additional or multiple data factors may exist for each singularmeasurement, or patient interaction.

Based upon the answers to the above questions, the quality of the CDRS-Rscore may be measured at each interaction point and the quality of thosemeasurements may be quantified within the methods of the invention. Thismay be accomplished by determining the actual data values for the datafactors for each interaction point (baseline, four weeks, eight weeks),obtaining actual data value scores by applying a scoring system to theactual data value scores, weighting the actual data value scoresaccording to their respective importance to data quality for eachinteraction point, adding the weighted actual data value scores for eachindividual at each interaction point to produce individual subjectsuitability scores (e.g., FIG. 3), comparing those scores topre-calculated ideal subject suitability scores (according to themethods described herein, e.g., FIGS. 1 & 2), and measuring thedeviation between the individual subject suitability scores and theideal subject suitability scores (e.g., FIG. 1).

Each critical safety or efficacy variable can be similarly broken downinto the data factors that determine the quality of the collected data.The following questions illustrate an example of criteria that may beutilized to define the quality of a discrete measurement underconsideration (and thus develop data factors for the methods ofinvention): 1) Was the data collected? 2) Was the data collected withinthe correct time interval as specified by the protocol? 3) Is theresulting data within the expected range for the particular variable(s)?Depending on the requirements for each clinical trial protocol, and thenature of the measurements being made, different additional factors maybe required to develop data factors that represent the issues of dataquality for a particular variable.

FIG. 4 is an exemplary illustration of ideal subject suitability scoresgenerated for five interaction points of a clinical trial usinginclusion/exclusion, safety, and efficacy data for a clinical trial. Theideal subject suitability scores are based upon the data requirementsfor each protocol directed interaction point. For example, during aninteraction point 4.1 (the enrollment visit), the ideal subjectsuitability score may be determined by the criteria listed in theinclusion and exclusion checklist and the baseline data required for thesafety and efficacy variables. For interaction points 4.2 through 4.5,the ideal subject suitability scores may be based on the detailed studycollection needs as found in the protocol that define the requirementsfor safety and efficacy data. Depending on the study and its datarequirements, there may be additional data factors that are consideredfor quality calculation with regard to a specific measurement orinteraction.

FIG. 4 illustrates one way the different categories (e.g.,inclusion/exclusion, safety, efficacy, or other categories) of datacomprising the overall ideal subject suitability score for aninteraction point may be identified and differentiated. Thisidentification and/or differentiation may relate to, or result from,indexing of data factors used in generation of ideal subject suitabilityscores (such as, for example the indexing performed in operation 203 ofFIG. 2). FIG. 4 also illustrates that the relative contribution of theconstituent data categories to that interaction point's ideal subjectsuitability score, may be also identified, visualized, and/or utilized.For example, for interaction point 4.1, inclusion/exclusion criteriacomprise about 100 points of the 275 point overall ideal subjectsuitability score. Safety criteria contribute 75 points to the idealsubject suitability score, while efficacy criteria contribute 100points.

FIG. 5 is an exemplary illustration according to an embodiment of theinvention, wherein ideal subject suitability scores are compared toseveral individual subject suitability scores over the course ofinteraction points 5.1 to 5.5. As detailed above, similar methods may beutilized to calculate the individual subject suitability scores (e.g.,SSS-1, SSS-2, etc.) as are used to determine the ideal subjectsuitability score (e.g., Ideal SSS). The deviation of individual subjectsuitability scores from the ideal subject suitability score for eachinteraction point results in a measure of the quality of the data foreach interaction point.

From the data displayed in FIG. 5, it is apparent that some subjects aremissing their study visits/interaction points (or the data is beingcollected with startling deviation from the protocol) and as such, aregiven a subject suitability scores of zero for that interaction point(e.g., SSS-1 for interaction point 5.2; SSS-5 for interaction points 5.3through 5.5). By reviewing the data for interaction point 5.1 (theenrollment visit), it is apparent that this study site is not enrollingsubjects that match the inclusion/exclusion criteria for the protocol,given the low subject suitability scores of SSS-1 and SSS-3. Insubsequent interaction points 5.2 through 5.5 the data quality forseveral subjects is unsatisfactory given the low individual subjectsuitability scores (e.g, SSS-3 for interaction points 5.2, 5.3, 5.4 and5.5). The precise range for what deviation in subject suitability scoresis acceptable may vary among clinical trials and/or interaction points.

As mentioned above, the degree of deviation in individual subjectsuitability scores, aggregate individual subject suitability scores, orstudy site aggregate scores may have a predictive value on the abilityof the investigator or site to perform the clinical trial. For example,if a study site or an investigator has too many actual subjects whoseindividual subject suitability scores deviate from the ideal subjectsuitability score, it is unlikely that these subjects will meet theclinical objectives of the protocol since the data will be of poorquality. On the contrary, if a study site has subjects whose individualsubject suitability score closely tracks the ideal subject suitabilityscore, it is likely that these subjects will meet the clinicalobjectives of the protocol and the data will be of high quality.

FIG. 6 is an exemplary illustration according to an embodiment of theinvention, wherein an ideal subject suitability score is compared toindividual subject suitability scores for a single interaction point.The ideal and individual subject suitability scores illustrated in FIG.6 are broken down by the categories of data factors utilized for thatparticular interaction point. In this case, the particular interactionpoint analyzed is the enrollment visit. As such, the data categoriesinclude the inclusion/exclusion data, the baseline safety data, and thebaseline efficacy data. Breaking down subject suitability scores, asillustrated in FIG. 6, may enable a systematic review of potentialproblem areas for this particular data point. In some embodiments, foranalysis or other purposes, the ideal, individual, and/or aggregatesubject suitability scores for one or more interaction points may bebroken down in different ways such as, for example, by individual datafactors, by individual subject interaction, or by other characteristics.

In viewing the example of FIG. 6, it is evident that the study site isinitially having difficulties in the enrollment of study subjects (asevidenced by the low inclusion/exclusion scores for ID-001 and ID-002).In this example, the lower portion of each bar in the bar graphrepresents a quality assessment of the inclusion and exclusion data. Thefirst two subjects enrolled (ID-001 and ID-002) are well below therequirement of the ideal subject and will not be considered suitablesubjects.

In this example, the topmost portion of each bar represents the baselinemeasurements to determine efficacy. As such, ID-001 and ID-002 also havescores that are well below the efficacy requirements for a suitablesubject. The precise range for what deviation in subject suitabilityscores is acceptable may vary among clinical trials. In this example 30out of 50 for efficacy data may be thought to be unacceptable.

It also appears that the exemplary study site illustrated in FIG. 6 isnot having difficulty performing the safety related measures. However,this will not be enough to offset the difficulties in meeting theinclusion/exclusion and efficacy criteria.

The illustration of the invention in FIG. 6 illustrates the fact thatthe invention enables real-time, prospective review of information.Thus, providing an “early warning” to the study sponsor about thequalifications, capabilities, and/or aptitude of the clinicalinvestigator, the study site, and/or the study protocol itself. In somecircumstances, the study sponsor may elect to close down the site. Inother circumstances, the study sponsor may choose to implement aremediation program such as, for example, re-training the site staff sothey increase their compliance with the study protocol. In the exampleof FIG. 6, a remediation program was initiated after the enrollment ofsubject ID-002. Subsequently enrolled subjects had an improved subjectsuitability score resulting in higher quality data.

Properly screening subjects for a clinical trial may be a critical firststep in ensuring that a study protocol will be executed correctly.Deciding whether a potential investigator and/or the study site willhave an acceptable patient population to fulfill the requirement of aprotocol is more difficult. Traditionally, when the sponsor of aclinical trial or its representative contract research organization(CRO) is identifying potential investigators or sites for a drug studythey often rely on a non-systematic approach for determining thecapability of the investigator or site to supply appropriate subjects.The net result is that sites that appeared promising often times do notdeliver the enrolled and completed study subjects as promised. In oneembodiment, the invention offers a more rigorous method for qualifyingstudy sites to participate in a drug study and to aid in the prospectiveevaluation of potential study subjects.

In one embodiment, as illustrated in FIG. 7, a method 700 is provided todetermine if the site has access to, or is capable of enrolling clinicaltrial subjects that are likely to produce high quality data. In anoperation 701, an ideal subject suitability score for an ideal subjectparticipant may be generated. This ideal subject suitability score maybe based on data factors that concentrate on exclusion and inclusioncriteria according to the protocol of the clinical trial. Thisformulation may involve operations similar to the development of idealsubject suitability scores as described in the methods above (e.g.,identification of data factors, indexing of data factors, ranking ofdata factors, associating scoring systems with data factors, applyingthe scoring systems, weighting the resultant ideal data factor scores,summing the weighted ideal data factor scores, and/or other operations).

In an operation 703, historical data regarding previous studiesconducted at the proposed site (which may be indicative of the site'sability to enroll subjects that produce high quality data), populationdata for the area surrounding the study site (which may be indicative ofthe presence of subjects capable of producing high quality data for theparticular clinical trial at issue), and/or other data may be gathered.In an operation 705, the data from operation 703 may be used to generatea study site aptitude score. The study site aptitude score may begenerated by identifying the actual historical, population, and/or otherdata values that corresponds to the data factors identified in theformula used to develop the ideal subject suitability score of operation701. This “identified data” may then be applied to theindex/scoring/weighting procedures used to develop the ideal subjectsuitability score. For example, application of scoring systemsassociated with the data factors to actual data values derived from the“identified data” may yield study site data scores. These study sitedata scores may be weighted and/or otherwise used to determine the studysite aptitude score. In an operation 707, a study site aptitudeassessment may be generated. The study site aptitude assessment mayreflect the study site's access to, and/or ability to enroll clinicaltrial subjects likely to yield high quality data. In one embodiment, thestudy site aptitude assessment may be generated by comparing the studysite aptitude score to the ideal subject suitability score. Thedeviation between the two scores may be used to determine the study siteaptitude assessment for the proposed study site.

In an embodiment illustrated in FIG. 8, a method 800 according to anembodiment of the invention is provided, wherein the acceptability of apotential subject as an entrant to the study may be determined prior tothe potential subject receiving therapy. In an operation 801, an idealsubject suitability score may be generated. Operation 801 may utilizeinclusion and exclusion criteria detailed in the clinical trial'sprotocol. The development of the ideal subject suitability score ofoperation 801 may involve operations similar to the development of idealsubject suitability scores as described in the methods above (e.g.,identification of data factors, indexing of data factors, ranking ofdata factors, associating scoring systems with the data factors,applying the scoring systems, weighting the resultant ideal data factorscores, summing weighted ideal data factor scores, and/or otheroperations).

In an operation 803, individual subject suitability scores may begenerated, prior to enrollment, by collecting actual inclusion/exclusiondata from potential subjects. This generation may involve operationssimilar to the development of individual subject suitability scores asdescribed in the methods above (e.g., correlating actual data values toidentified data factors, ranking actual data values, applying scoringsystems, weighting resultant actual data factor scores, summing weightedactual data factor scores, and/or other operations). In an operation805, a subject suitability assessment may be generated. The subjectsuitability assessment may reflect an individual potential clinicaltrial subject's potential to produce high quality data. In someembodiments, the subject suitability assessment may be generated bycomparing an individual potential subject's individual subjectsuitability score to the ideal subject suitability scores. As such, theaptitude of individual potential subjects (or groups thereof) may bejudged.

Clinical trials of new drugs can often fail in late stage development,despite success at earlier phases. Given the expense of repeating alarge clinical trial, a method by which a completed trial could besystematically reviewed may aid a drug or medical device sponsor indeciding whether or not they should commit to performing another largemulti-center trial. In one embodiment of the invention illustrated inFIG. 9, a process 900 for evaluating a completed trial is provided. Inan operation 901, the protocol of a completed clinical trial may beutilized to generate ideal subject suitability scores for one or moreinteraction points of the completed clinical trial. This scoregeneration may involve operations similar to the development of idealsubject suitability scores as described in the methods above (e.g.,identification of data factors, indexing of data factors, ranking ofdata factors, associating scoring systems with the data factors,applying the scoring systems, weighting resultant ideal data factorscores, summing weighted data factor scores, and/or other operations).

In an operation 903, individual retrospective subject suitability scoresmay be generated using available subject interaction data. Thisdevelopment may involve operations similar to the development ofindividual subject suitability scores as described in the methods above(e.g., correlating actual data values to identified data factors,ranking actual data values, applying scoring systems, weightingresultant actual data factor scores, summing weighted actual data factorscores, and/or other operations). In an operation 905, the individualscores may be compared to the ideal scores to assess the quality of thedata acquired from the completed clinical trial. Process 900 may enablevarious selective/differential groupings and comparisons between idealand individual scores that may be utilized for data mining of thequality of data for a completed clinical trial. This data mining mayenable identification of one or more characteristics of the completedclinical trial, including, for example, elements that caused the failureof the completed clinical trial.

Those having skill in the art will appreciate that the processes of theinvention described herein may work with their constituent operationsperformed in varying orders. Accordingly, some or all of the operationsdescribed herein may be used in various combinations to perform theprocesses of the invention.

According to an embodiment of the invention illustrated in FIG. 10, theinvention provides a computer-implemented system 1000 that enablesperformance of the data quality assessment features and other featuresdescribed herein. Computer implemented system 1000 may include acomputer system 1001, a data quality application 1003, one or moresoftware modules 1005 a-n, a data storage devices 1007 a-n, one or moreterminal devices 1009 a-n, and/or other elements.

Computer system 1001 may include one or more personal computers, laptopcomputers, servers, or other machines which may be or include, forinstance, a workstation running Microsoft Windows™ NT™, MicrosoftWindows™ 2000, Unix, Linux, Xenix, IBM, AIX™, Hewlett-Packard UX™,Novell Netware™, Sun Microsystems Solaris™, OS/2™, BeOS™, Mach, Apache,OpenStep™, or other operating system or platform. Computer system 1001may include one or more processors 1011 which may receive, send, and/ormanipulate data for the performance of the features, functions, and oroperations of the invention as described herein, including the any orall of the operations of the methods described in FIGS. 1, 2, 3, 7, 8, 9and/or other methods.

According to one embodiment, computer system 1001 may host a dataquality application 1003. Data quality application 1003 may comprise acomputer application maintained by an clinical trial sponsor, a clinicaltrial administrator, a research services provider, or other entity.

According to an embodiment of the invention, data quality application1003 may include or comprise one or more software modules 1005 a-n forgenerating ideal subject suitability scores, generating individualsubject suitability scores, analyzing clinical trial protocols,identifying data factors, correlating data values (actual or ideal) toidentified data factors, indexing data factors or data values, rankingdata factors or data values, devising scoring systems, associatingscoring systems with data factors, applying scoring systems to datavalues (actual or ideal), producing data factor scores (actual orideal), weighting data factor scores (actual or ideal), summing datafactor scores (actual or ideal), grouping and/or averaging ideal and/orindividual subject suitability scores, comparing ideal and individualscores, assessing data quality, generating data quality assessments(including study site aptitude assessments, subject suitabilityassessments, and/or other assessments) mining data quality information,generating remedial measures, or for performing any of the otheroperations described in detail herein.

In particular, data quality application 1003 may include an idealsubject suitability score module 1005 a. Ideal subject suitability scoremodule 1005 a may enable the performance of operations for thegeneration of ideal subject suitability scores for one or moreinteraction points of a clinical trial according to the protocol of theclinical trial (including the operations detailed in FIGS. 2, 7, 8,and/or 9).

Data quality application 1003 may also include individual subjectsuitability score module 1005 b. Individual subject suitability scoremodule 1005 b may enable the performance of operations for thedevelopment of one or more individual subject suitability scoresaccording to data received from actual subject interaction with one ormore interaction points of a clinical trial (including the operationsdetailed in FIGS. 3, 8 and 9).

Data quality application 1003 may also include a quality assessmentmodule 1005 c, which may enable the generation of a data qualityassessment. Quality assessment module 1005 c may also enable theperformance of operations for the aggregation, averaging, and/orgrouping of individual subject suitability scores and/or ideal subjectsuitability scores. Quality assessment module 1005 c may also enable theperformance of operations for the comparison of ideal and individualsubject suitability scores. Quality assessment module 1005 c may alsoenable the performance of operations for the measurement of deviationsbetween ideal and individual subject suitability scores. Qualityassessment module 1005 c may enable operations for the qualityassessment of data obtained from patient interaction with one or moreinteraction points of a clinical trial.

Data quality application 1003 may also include a data mining module 1005d. Data mining module 1005 d may enable operations for the selectiveand/or differential assessment of clinical trial data and theidentification, generation, and/or implementation of remedial measures.Data mining module may also enable retrospective analysis of individualand ideal subject suitability scores of a previously performed clinicaltrial for to identify one or more elements that may have caused thefailure of the previously performed trial.

In some embodiments, data quality application may include an aptitudeassessment module for generating a study site aptitude assessment. Inother embodiments, data quality application may include a subjectsuitability assessment module for generating a subject suitabilityassessment.

Other features of the invention, including features described above maybe enabled by other modules included in data quality application 1003.One or more of the modules included in data quality application 1003 maybe combined. For some purposes, not all modules may be necessary.

In some embodiments, computer system 1001 may be operatively connectedto one or more data storage devices 1007 a-n. Data storage devices 1007a-n may be utilized to store any of the data utilized by or produced byany of the processes or functions described herein. Data storage devices1007 a-n may be, include, or interface to, for example, an Oracle™relational database sold commercially by Oracle Corporation. Otherdatabases, such as Informix™, DB2 (Database 2) or other data storage orquery formats, platforms, or resources such as OLAP (On Line AnalyticalProcessing), SQL (Standard Language Query), a SAN (storage areanetwork), Microsoft Access™ or others may also be used, incorporated, oraccessed into the invention.

In one embodiment, computer system 1001 may be operatively connected toone or more terminal devices 109 a-n. This operative connection mayoccur over a network (e.g., the Internet) or other operative connection.Communication between computer system 1001 and one or more terminaldevices 1011 a-n may be utilized to transmit data necessary for theimplementation of the processes or functions of the invention such as,for example, entry of patient interaction data from a remote terminaldevice 1011 at a clinical trial study site for transmission to a centralprocessing site.

One or more terminal devices 1011 a-n may include a personal computer, aserver, a laptop computer, a personal digital assistant (PDA), or otherdevice. In some embodiment, one or more terminal devices 1011 a-n mayinclude a wireless terminal device.

Those having skill in the art will appreciate that the inventiondescribed herein may work with various system configurations.Accordingly, more or less of the aforementioned system components may beused and/or combined in various embodiments. It should also beunderstood that various software modules 1005 a-n and data qualityapplication 1003 that are utilized to accomplish the functionalitiesdescribed herein may be maintained on one or more of computer system1001, processors 1002, terminal devices 1011 a-n or other components ofsystem 1000, as necessary. In other embodiments, as would beappreciated, the functionalities described herein may be implemented invarious combinations of hardware and/or firmware, in addition to, orinstead of, software.

In one embodiment, the invention may include a computer readable mediumcontaining instructions that, when executed by at least one processor(such as, for example processor 1011 of system 1000), cause the at leastone processor to enable and/or perform the features, functions, and oroperations of the invention as described herein, including the any orall of the operations of the processes described in FIGS. 1, 2, 3, 7, 8,9, and/or other operations.

While the computer readable medium and computer implemented systemdetailed above may be utilized for performing the methods of theinvention, in some embodiments, some or all of the operations or methodsof the invention may be performed manually.

Other embodiments, uses and advantages of the invention will be apparentto those skilled in the art from consideration of the specification andpractice of the invention disclosed herein. The specification should beconsidered exemplary only, and the scope of the invention is accordinglyintended to be limited only by the following claims.

1-36. (canceled)
 37. A method for assessing quality of clinical trialdata generated during a clinical trial, wherein the clinical trialincludes one or more clinical trial subjects that each interact with aplurality of data collection points of the clinical trial, the methodcomprising: generating, by a computer processor, an ideal data qualityscore for clinical trial data collected at a data collection point,wherein the data collection point comprises a point during the clinicaltrial at which clinical trial data for the clinical trial is collected,wherein the clinical trial is conducted according to a clinical trialprotocol that includes a protocol by which the clinical trial datashould be collected at the data collection point, and wherein the idealdata quality score represents a benchmark indicative of the clinicaltrial data having been collected in adherence to the clinical trialprotocol at the data collection point; receiving, by the computerprocessor, an indication of an actual procedure by which actual clinicaltrial data was collected during the data collection point, wherein theactual clinical trial data is associated with a clinical trial subjectundergoing the clinical trial; generating, by the processor, anindividual data quality score based on one or more data factorsassociated with the protocol by which the clinical trial data should becollected, wherein the one or more data factors are different fordifferent protocols by which the clinical trial data should becollected, and wherein the one or more data factors are used todetermine a level of compliance with the protocol by which the clinicaltrial data should be collected such that the individual data qualityscore represents whether the actual procedure by which the actualclinical trial data was collected from the clinical trial subject at thedata collection point is in adherence with the protocol by which theclinical trial data should be collected during the data collectionpoint; generating, by the processor, a data quality assessment based ona comparison of the individual data quality score and the ideal dataquality score, the data quality assessment representative of the degreeto which the protocol by which the clinical trial data should becollected was adhered to during the clinical trial; and providing thedata quality assessment to a decision maker so that the decision makercan accept or reject the clinical trial data for the clinical trialbased on the data quality assessment.
 38. The method of claim 37,wherein the ideal data quality score is based on the one or more datafactors.
 39. The method of claim 38, wherein the one or more datafactors include data factors relevant to subject inclusion data, subjectexclusion data, efficacy data, or safety data.
 40. The method of claim38, wherein generating an ideal data quality score further comprises:associating a scoring system with each of the one or more data factors;generating one or more ideal data factor scores by applying the scoringsystem associated with a data factor to an ideal data value of the datafactor; and determining the ideal data quality score based on the one ormore ideal data factor scores.
 41. The method of claim 38, whereingenerating an ideal data quality score further comprises: indexing theone or more data factors according to one or more characteristics;associating a scoring system with each of the one or more data factors;generating one or more ideal data factor scores by applying the scoringsystem of a data factor to an ideal data value of the data factor;weighting the one or more ideal data factor scores; and determining theideal data quality score based on the weighted ideal data factor scoresof the one or more data factors, wherein the ideal data quality score isdetermined by summing the weighted ideal data factor scores of each ofthe one or more data factors.
 42. The method of claim 38, whereingenerating an individual data quality score further comprises:generating one or more actual data factor scores by applying a scoringsystem to each of one or more actual data values collected from theinteraction of the at least one individual clinical trial subject withthe data collection point, wherein the one or more actual data valuescorrespond to actual data collected for each of the one more datafactors on which the individual data quality score is based; anddetermining the individual data quality score based on the one or moreactual data factor scores.
 43. The method of claim 38, whereingenerating an individual data quality score further comprises: indexingone or more actual data values according to one or more characteristics,wherein the one or more actual data values are collected from theinteraction of the subject with the data collection point, and whereinthe one or more actual data values correspond to actual data collectedfor each of the one more data factors on which the ideal data qualityscore is based; generating one or more actual data factor scores byapplying a scoring system to each of one or more actual data values;weighting the one or more actual data factor scores; and determining theindividual data quality score based on the one or more weighted actualdata factor scores, wherein the individual data quality score isdetermined by summing the one or more weighted actual data factorscores.
 44. The method of claim 37, further comprising generating one ormore remedial measures to improve the data quality for the datacollection point based on the data quality assessment.
 45. A method forassessing quality of clinical trial data generated during a clinicaltrial, wherein the clinical trial includes one or more clinical trialsubjects that each interact with a plurality of data collection pointsof the clinical trial, the method comprising: generating, by a computerprocessor, an ideal data quality score for clinical trial data collectedat a data collection point, wherein the data collection point comprisesa point during the clinical trial at which the clinical trial data forthe clinical trial is collected, wherein the clinical trial is conductedaccording to a clinical trial protocol that includes a protocol by whichthe clinical trial data should be collected at the data collectionpoint, and wherein the ideal data quality score represents a benchmarkindicative of the clinical trial data having been collected in adherenceto the clinical trial protocol at the data collection point, and whereinthe ideal data quality score for the data collection point is generatedby: indexing, by the processor, one or more data factors that arerelevant to the data collection point, wherein the one or more datafactors are different for different protocols by which the clinicaltrial data should be collected at the data collection point,associating, by the processor, a scoring system with each of the one ormore data factors, generating, by the processor, one or more ideal datafactor scores by applying the scoring system associated with a datafactor to an ideal data value of the data factor, weighting, by theprocessor, the one or more ideal data factor scores, and determining, bythe processor, the ideal data quality score based on the weighted idealdata factor scores, wherein the ideal data quality score is determinedby summing the weighted ideal data factor scores of each of the one ormore data factors; generating, by the processor, an individual dataquality score based on one or more data factors associated with theprotocol by which the clinical trial data should be collected, whereinthe one or more data factors are different for different protocols bywhich the clinical trial data should be collected, and wherein the oneor more data factors are used to determine a level of compliance withthe protocol by which the clinical trial data should be collected suchthat the individual data quality score represents whether the actualprocedure by which the actual clinical trial data was collected from theclinical trial subject at the data collection point is in adherence withthe protocol by which the clinical trial data should be collected duringthe data collection point, wherein the individual data quality score isgenerated by: indexing, by the processor, one or more actual data valuesthat are collected from an interaction of the clinical trial subjectwith the data collection point, and wherein the one or more actual datavalues correspond to actual data collected for each of the one or moredata factors on which the ideal data quality score is based, generating,by the processor, one or more actual data factor scores by applying thescoring system associated with each individual data factor to an actualdata value of the data factor, weighting, by the processor, the actualdata factor scores according to the weighted value of theircorresponding ideal data factor scores, and determining, by theprocessor, the individual data quality score based on the one or moreweighted actual data factor scores, wherein the individual data qualityscore is determined by summing the one or more weighted actual datafactor scores and wherein the one or more actual data factor scores areused to determine a level of compliance with the clinical trial protocolsuch that the individual data quality score represents whether an actualprocedure by which the clinical trial data was collected from theclinical trial subject at the data collection point is in adherence tothe clinical trial protocol; and generating, by the processor, a dataquality assessment based on a comparison of the individual data qualityscore and the ideal data quality score, wherein the data qualityassessment is representative of the degree to which the protocol bywhich the clinical trial data should be collected was adhered to duringthe clinical trial.
 46. A computer-implemented system that assessesquality of clinical trial data generated during a clinical trial,wherein the clinical trial includes one or more clinical trial subjectsthat each interact with a plurality of data collection point of theclinical trial, the system comprising: one or more computer processorsconfigured to: generate an ideal data quality score for clinical trialdata collected at a data collection point, wherein the data collectionpoint comprises a point during the clinical trial at which clinicaltrial data for the clinical trial is collected, wherein the clinicaltrial is conducted according to a clinical trial protocol that includesa protocol by which the clinical trial data should be collected at thedata collection point, and wherein the ideal data quality scorerepresents a benchmark indicative of the clinical trial data having beencollected in adherence to the clinical trial protocol at the datacollection point; receive an indication of an actual procedure by whichactual clinical trial data was collected during the data collectionpoint, wherein the actual clinical trial data is associated with aclinical trial subject undergoing the clinical trial; generate anindividual data quality score based on one or more data factorsassociated with the protocol by which the clinical trial data should becollected, wherein the one or more data factors are different fordifferent protocols by which the clinical trial data should becollected, and wherein the one or more data factors are used todetermine a level of compliance with the protocol by which the clinicaltrial data should be collected such that the individual data qualityscore represents whether the actual procedure by which the actualclinical trial data was collected from the clinical trial subject at thedata collection point is in adherence with the protocol by which theclinical trial data should be collected during the data collectionpoint; generate a data quality assessment based on a comparison of theindividual data quality score and the ideal data quality score, the dataquality assessment representative of the degree to which the protocol bywhich the clinical trial data should be collected was adhered to duringthe clinical trial; and provide the data quality assessment to adecision maker so that the decision maker can decide whether to acceptor reject the clinical trial data for the clinical trial based on thedata quality assessment.
 47. The system of claim 46, wherein the idealdata quality score is based on the one or more data factors.
 48. Thesystem of claim 47, wherein the one or more data factors include datafactors relevant to subject inclusion data, subject exclusion data,efficacy data, or safety data.
 49. The system of claim 47, whereingenerate an ideal data quality score comprises: associate a scoringsystem with each of the one or more data factors; generate one or moreideal data factor scores by applying the scoring system associated witha data factor to an ideal data value of the data factor; and determinethe ideal data quality score based on the one or more ideal data factorscores.
 50. The system of claim 47, wherein generate an ideal dataquality score comprises: index the one or more data factors according toone or more characteristics; associate a scoring system with each of theone or more data factors; generate one or more ideal data factor scoresby applying the scoring system of a data factor to an ideal data valueof the data factor; weight the one or more ideal data factor scores; anddetermine the ideal data quality score based on the weighted ideal datafactor scores of the one or more data factors, wherein the ideal dataquality score is determined by summing the weighted ideal data factorscores of each of the one or more data factors.
 51. The system of claim47, wherein generate an individual data quality score comprises:generate one or more actual data factor scores by applying a scoringsystem to each of one or more actual data values collected from theinteraction of the at least one individual clinical trial subject withthe data collection point, wherein the one or more actual data valuescorrespond to actual data collected for each of the one more datafactors on which the individual data quality score is based; anddetermine the individual data quality score based on the one or moreactual data factor scores.
 52. The system of claim 47, wherein generatean individual data quality score comprises: index one or more actualdata values according to one or more characteristics, wherein the one ormore actual data values are collected from the interaction of thesubject with the data collection point, and wherein the one or moreactual data values correspond to actual data collected for each of theone more data factors on which the ideal data quality score is based;generate one or more actual data factor scores by applying a scoringsystem to each of one or more actual data values; weight the one or moreactual data factor scores; and determine the individual data qualityscore based on the one or more weighted actual data factor scores,wherein the individual data quality score is determined by summing theone or more weighted actual data factor scores.
 53. The system of claim46, wherein the one or more computer processors are further configuredto generate one or more remedial measures to improve the data qualityfor the data collection point based on the data quality assessment.